{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorial 01: Running Sumo Simulations\n",
    "\n",
    "This tutorial walks through the process of running non-RL traffic simulations in Flow. Simulations of this form act as non-autonomous baselines and depict the behavior of human dynamics on a network. Similar simulations may also be used to evaluate the performance of hand-designed controllers on a network. This tutorial focuses primarily on the former use case, while an example of the latter may be found in `tutorial09_controllers.ipynb`.\n",
    "\n",
    "In this tutorial, we simulate a initially perturbed single lane ring road. We witness in simulation that as time advances the initially perturbations do not dissipate, but instead propagates and expands until vehicles are forced to periodically stop and accelerate. For more information on this behavior, we refer the reader to the following article [1].\n",
    "\n",
    "## 1. Components of a Simulation\n",
    "All simulations, both in the presence and absence of RL, require two components: a *network*, and an *environment*. Networks describe the features of the transportation network used in simulation. This includes the positions and properties of nodes and edges constituting the lanes and junctions, as well as properties of the vehicles, traffic lights, inflows, etc. in the network. Environments, on the other hand, initialize, reset, and advance simulations, and act the primary interface between the reinforcement learning algorithm and the network. Moreover, custom environments may be used to modify the dynamical features of an network.\n",
    "\n",
    "## 2. Setting up a Network\n",
    "Flow contains a plethora of pre-designed networks used to replicate highways, intersections, and merges in both closed and open settings. All these networks are located in `flow/networks`. In order to recreate a ring road network, we begin by importing the network `RingNetwork`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from flow.networks.ring import RingNetwork"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This network, as well as all other networks in Flow, is parametrized by the following arguments: \n",
    "* name\n",
    "* vehicles\n",
    "* net_params\n",
    "* initial_config\n",
    "* traffic_lights\n",
    "\n",
    "These parameters allow a single network to be recycled for a multitude of different network settings. For example, `RingNetwork` may be used to create ring roads of variable length with a variable number of lanes and vehicles.\n",
    "\n",
    "### 2.1 Name\n",
    "The `name` argument is a string variable depicting the name of the network. This has no effect on the type of network created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "name = \"ring_example\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 VehicleParams\n",
    "The `VehicleParams` class stores state information on all vehicles in the network. This class is used to identify the dynamical behavior of a vehicle and whether it is controlled by a reinforcement learning agent. Morover, information pertaining to the observations and reward function can be collected from various get methods within this class.\n",
    "\n",
    "The initial configuration of this class describes the number of vehicles in the network at the start of every simulation, as well as the properties of these vehicles. We begin by creating an empty `VehicleParams` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from flow.core.params import VehicleParams\n",
    "\n",
    "vehicles = VehicleParams()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once this object is created, vehicles may be introduced using the `add` method. This method specifies the types and quantities of vehicles at the start of a simulation rollout. For a description of the various arguements associated with the `add` method, we refer the reader to the following documentation ([VehicleParams.add](https://flow.readthedocs.io/en/latest/flow.core.html?highlight=vehicleparam#flow.core.params.VehicleParams)).\n",
    "\n",
    "When adding vehicles, their dynamical behaviors may be specified either by the simulator (default), or by user-generated models. For longitudinal (acceleration) dynamics, several prominent car-following models are implemented in Flow. For this example, the acceleration behavior of all vehicles will be defined by the Intelligent Driver Model (IDM) [2]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from flow.controllers.car_following_models import IDMController"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another controller we define is for the vehicle's routing behavior. For closed network where the route for any vehicle is repeated, the `ContinuousRouter` controller is used to perpetually reroute all vehicles to the initial set route."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from flow.controllers.routing_controllers import ContinuousRouter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we add 22 vehicles of type \"human\" with the above acceleration and routing behavior into the `Vehicles` class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vehicles.add(\"human\",\n",
    "             acceleration_controller=(IDMController, {}),\n",
    "             routing_controller=(ContinuousRouter, {}),\n",
    "             num_vehicles=22)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 NetParams\n",
    "\n",
    "`NetParams` are network-specific parameters used to define the shape and properties of a network. Unlike most other parameters, `NetParams` may vary drastically depending on the specific network configuration, and accordingly most of its parameters are stored in `additional_params`. In order to determine which `additional_params` variables may be needed for a specific network, we refer to the `ADDITIONAL_NET_PARAMS` variable located in the network file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from flow.networks.ring import ADDITIONAL_NET_PARAMS\n",
    "\n",
    "print(ADDITIONAL_NET_PARAMS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Importing the `ADDITIONAL_NET_PARAMS` dictionary from the ring road network, we see that the required parameters are:\n",
    "\n",
    "* **length**: length of the ring road\n",
    "* **lanes**: number of lanes\n",
    "* **speed**: speed limit for all edges\n",
    "* **resolution**: resolution of the curves on the ring. Setting this value to 1 converts the ring to a diamond.\n",
    "\n",
    "\n",
    "At times, other inputs may be needed from `NetParams` to recreate proper network features/behavior. These requirements can be found in the network's documentation. For the ring road, no attributes are needed aside from the `additional_params` terms. Furthermore, for this tutorial, we use the network's default parameters when creating the `NetParams` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from flow.core.params import NetParams\n",
    "\n",
    "net_params = NetParams(additional_params=ADDITIONAL_NET_PARAMS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4 InitialConfig\n",
    "\n",
    "`InitialConfig` specifies parameters that affect the positioning of vehicle in the network at the start of a simulation. These parameters can be used to limit the edges and number of lanes vehicles originally occupy, and provide a means of adding randomness to the starting positions of vehicles. In order to introduce a small initial disturbance to the system of vehicles in the network, we set the `perturbation` term in `InitialConfig` to 1m."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from flow.core.params import InitialConfig\n",
    "\n",
    "initial_config = InitialConfig(spacing=\"uniform\", perturbation=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.5 TrafficLightParams\n",
    "\n",
    "`TrafficLightParams` are used to describe the positions and types of traffic lights in the network. These inputs are outside the scope of this tutorial, and instead are covered in `tutorial10_traffic_lights.ipynb`. For our example, we create an empty `TrafficLightParams` object, thereby ensuring that none are placed on any nodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from flow.core.params import TrafficLightParams\n",
    "\n",
    "traffic_lights = TrafficLightParams()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Setting up an Environment\n",
    "\n",
    "Several envionrments in Flow exist to train autonomous agents of different forms (e.g. autonomous vehicles, traffic lights) to perform a variety of different tasks. These environments are often network- or task-specific; however, some can be deployed on an ambiguous set of networks as well. One such environment, `AccelEnv`, may be used to train a variable number of vehicles in a fully observable network with a *static* number of vehicles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from flow.envs.ring.accel import AccelEnv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although we will not be training any autonomous agents in this tutorial, the use of an environment allows us to view the cumulative reward simulation rollouts receive in the absence of autonomy.\n",
    "\n",
    "Although we will not be training any autonomous agents in this exercise, the use of an environment allows us to view the cumulative reward simulation rollouts receive in the absence of autonomy.\n",
    "\n",
    "Envrionments in Flow are parametrized by several components, including the following attributes:\n",
    "* `sim_params`\n",
    "* `env_params`\n",
    "* `network`\n",
    "* `net_params`\n",
    "* `initial_config`\n",
    "* `network`\n",
    "* `simulator`\n",
    "\n",
    "where `sim_params`, `env_params`, and `network` are the primary parameters of an environment. For the full list of attributes, please check `class Env` in `flow/envs/base.py`.\n",
    "\n",
    "Sumo envrionments in Flow are parametrized by three components:\n",
    "* `SumoParams`\n",
    "* `EnvParams`\n",
    "* `Network`\n",
    "\n",
    "### 3.1 SumoParams\n",
    "`SumoParams` specifies simulation-specific variables (e.g. `SumoParams` and `AimsunParams` are the variables related to SUMO and Aimsun simulator, respectively). These variables maay include the length a simulation step (in seconds), whether to render the GUI when running the experiment, and other variables. For this example, we consider a SUMO simulation, step length of 0.1s, and activate the GUI.\n",
    "\n",
    "Another useful parameter is `emission_path`, which is used to specify the path where the emissions output will be generated. They contain a lot of information about the simulation, for instance the position and speed of each car at each time step. If you do not specify any emission path, the emission file will not be generated. More on this in Section 5.\n",
    "\n",
    "### 3.1 SumoParams\n",
    "`SumoParams` specifies simulation-specific variables. These variables include the length a simulation step (in seconds) and whether to render the GUI when running the experiment. For this example, we consider a simulation step length of 0.1s and activate the GUI.\n",
    "\n",
    "Another useful parameter is `emission_path`, which is used to specify the path where the emissions output will be generated. They contain a lot of information about the simulation, for instance the position and speed of each car at each time step. If you do not specify any emission path, the emission file will not be generated. More on this in Section 5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from flow.core.params import SumoParams\n",
    "\n",
    "sim_params = SumoParams(sim_step=0.1, render=True, emission_path='data')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 EnvParams\n",
    "\n",
    "`EnvParams` specify environment and experiment-specific parameters that either affect the training process or the dynamics of various components within the network. Much like `NetParams`, the attributes associated with this parameter are mostly environment-specific, and can be found in the environment's `ADDITIONAL_ENV_PARAMS` dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from flow.envs.ring.accel import ADDITIONAL_ENV_PARAMS\n",
    "\n",
    "print(ADDITIONAL_ENV_PARAMS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Importing the `ADDITIONAL_ENV_PARAMS` variable, we see that it consists of only one entry, \"target_velocity\", which is used when computing the reward function associated with the environment. We use this default value when generating the `EnvParams` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from flow.core.params import EnvParams\n",
    "\n",
    "env_params = EnvParams(additional_params=ADDITIONAL_ENV_PARAMS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Setting up and Running the Experiment\n",
    "Once the inputs to the network and environment classes are ready, we are ready to set up a `Experiment` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from flow.core.experiment import Experiment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This object may be used to simulate rollouts in the absence of reinforcement learning agents, as well as acquire behaviors and rewards that may be used as a baseline with which to compare the performance of the learning agent. In this case, we choose to run our experiment for one rollout consisting of 3000 steps (300 s).\n",
    "\n",
    "**Note**: When executing the below code, remeber to click on the    <img style=\"display:inline;\" src=\"img/play_button.png\"> Play button after the GUI is rendered."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "flow_params = dict(\n",
    "    exp_tag='ring_example',\n",
    "    env_name=AccelEnv,\n",
    "    network=RingNetwork,\n",
    "    simulator='traci',\n",
    "    sim=sim_params,\n",
    "    env=env_params,\n",
    "    net=net_params,\n",
    "    veh=vehicles,\n",
    "    initial=initial_config,\n",
    "    tls=traffic_lights,\n",
    ")\n",
    "\n",
    "# number of time steps\n",
    "flow_params['env'].horizon = 3000\n",
    "exp = Experiment(flow_params)\n",
    "\n",
    "# run the sumo simulation\n",
    "_ = exp.run(1, convert_to_csv=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see from the above simulation, the initial perturbations in the network instabilities propogate and intensify, eventually leading to the formation of stop-and-go waves after approximately 180s.\n",
    "\n",
    "## 5. Visualizing Post-Simulation\n",
    "\n",
    "Once the simulation is done, a .xml file will be generated in the location of the specified `emission_path` in `SumoParams` (assuming this parameter has been specified) under the name of the network. In our case, this is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "emission_location = os.path.join(exp.env.sim_params.emission_path, exp.env.network.name)\n",
    "print(emission_location + '-emission.xml')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The .xml file contains various vehicle-specific parameters at every time step. This information is transferred to a .csv file if the `convert_to_csv` parameter in `exp.run()` is set to True. This file looks as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "pd.read_csv(emission_location + '-emission.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, each row contains vehicle information for a certain vehicle (specified under the *id* column) at a certain time (specified under the *time* column). These information can then be used to plot various representations of the simulation, examples of which can be found in the `flow/visualize` folder."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Modifying the Simulation\n",
    "This tutorial has walked you through running a single lane ring road experiment in Flow. As we have mentioned before, these simulations are highly parametrizable. This allows us to try different representations of the task. For example, what happens if no initial perturbations are introduced to the system of homogenous human-driven vehicles?\n",
    "\n",
    "```\n",
    "initial_config = InitialConfig()\n",
    "```\n",
    "\n",
    "In addition, how does the task change in the presence of multiple lanes where vehicles can overtake one another?\n",
    "\n",
    "```\n",
    "net_params = NetParams(\n",
    "    additional_params={\n",
    "        'length': 230, \n",
    "        'lanes': 2, \n",
    "        'speed_limit': 30, \n",
    "        'resolution': 40\n",
    "    }\n",
    ")\n",
    "```\n",
    "\n",
    "Feel free to experiment with all these problems and more!\n",
    "\n",
    "## Bibliography\n",
    "[1] Sugiyama, Yuki, et al. \"Traffic jams without bottlenecks—experimental evidence for the physical mechanism of the formation of a jam.\" New journal of physics 10.3 (2008): 033001.\n",
    "\n",
    "[2] Treiber, Martin, Ansgar Hennecke, and Dirk Helbing. \"Congested traffic states in empirical observations and microscopic simulations.\" Physical review E 62.2 (2000): 1805."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
